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LONG-TERM  GOALS 

The  long-term  goals  of  this  effort  are  to  provide  reliable  performance  prediction  and  accurate  system 
simulation  capabilities  for  underwater  electro-optical  identification  (EOID)  systems.  The  Electro- 
Optical  Detection  Simulator  (EODES)  suite  of  numerical  models  developed  under  this  program  will 
predict  the  impact  of  environmental  conditions,  system  parameters  (e.g.,  apertures  and  PMT  gains), 
and  operational  settings  (e.g.,  platform  speed  and  altitude)  on  system  performance.  The  models  are 
designed  to  support  mine  countermeasures  (MCM)  mission  planning,  system  design  analysis,  and 
operator  training.  It  is  anticipated  that  the  performance  prediction  models  will  also  be  integrated  into 
Tactical  Decision  Aids  (TDAs),  specifically  the  Mine  Warfare  Environmental  Decision  Aid  Library 
(MEDAL)  and/or  the  Environmental  Post  Mission  Analysis  (EPMA)  station. 

The  two  most  prominent  sensor  technologies  in  this  area  are  Laser  Line  Scan  (LLS)  and  Streak  Tube 
Imaging  Lidar  (STIL).  Examples  of  systems  using  these  technologies  are  the  AN/AQS-24  (using  LLS) 
and  a  variant  of  the  AN/AQS-20  (using  STIL)  mine-hunting  systems.  High-fidelity  models  for  these 
systems  have  already  been  incorporated  into  the  EODES  software.  The  EODES  LLS  model  is 
currently  undergoing  quality  assurance  verification  preliminary  to  its  inclusion  as  a  Navy  Standard 
model  in  the  Oceanographic  and  Atmospheric  Master  Library  (OAML).  The  STIL  model  will  be 
submitted  to  OAML  upon  further  validation.  The  Airborne  Laser  Mine  Detection  System  (ALMDS) 
also  uses  a  STIL  sensor  to  image  underwater  objects  from  an  airborne  platform.  An  ALMDS  model 
will  also  be  fully  integrated  into  EODES  after  further  development  and  validation. 

OBJECTIVES 

Our  objectives  are  to  develop  and  validate  EOID  models  to  compute  reliable  metrics  for  the  prediction 
of  system/operator  performance,  and  for  generating  synthetic  images  of  bottom  scenes  under  given 
environmental  conditions  and  operational  settings.  These  objectives  necessitated  the  development  of 
high-fidelity  physical  and  numerical  models  to  solve  the  radiative  transfer  equation  and  to  accurately 
calculate  the  time-dependent  laser  backscatter  return.  The  simulation  and  performance  prediction 
models  for  underwater  EOID  systems  are  complete  and  are  now  being  certified  for  ultimate 
incorporation  into  the  Navy  Standard  Oceanographic  and  Atmospheric  Master  Library  (OAML).  All 
EODES  code  is  written  in  ANSI  C/C++  (and  made  ISO  compliant)  to  ensure  portability  across 
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Public  reporting  burden  for  the  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and 
maintaining  the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information, 
including  suggestions  for  reducing  this  burden,  to  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports,  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington 
VA  22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  a  penalty  for  failing  to  comply  with  a  collection  of  information  if  it 
does  not  display  a  currently  valid  0MB  control  number. 

1.  REPORT  DATE 

2QQg  2.  REPORT  TYPE 

4.  TITLE  AND  SUBTITLE 

A  Comprehensive  Model  for  Performance  Prediction  of  Electro-Optical 
Systems 

6.  AUTHOR(S) 


7.  PEREORMING  ORGANIZATION  NAME(S)  AND  ADDRESS(ES) 

Metron,  Inc., 11911  Freedom  Dr.,  Suite  800,Reston,VA,20190-5602 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 


12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited 

13.  SUPPLEMENTARY  NOTES 

14.  ABSTRACT 

15.  SUBJECT  TERMS 


16.  SECURITY  CLASSIEICATION  OE: 

17.  LIMITATION  OE 

18.  NUMBER 

19a.  NAME  OE 

ABSTRACT 

OE  PAGES 

RESPONSIBLE  PERSON 

a.  REPORT 

unclassified 

b.  ABSTRACT 

unclassified 

c.  THIS  PAGE 

unclassified 

Same  as 
Report  (SAR) 

8 

3.  DATES  COVERED 

00-00-2008  to  00-00-2008 

5a.  CONTRACT  NUMBER 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

5d.  PROJECT  NUMBER 

5e.  TASK  NUMBER 

5f.  WORK  UNIT  NUMBER 

8.  PEREORMING  ORGANIZATION 
REPORT  NUMBER 

10.  SPONSOR/MONITOR’S  ACRONYM(S) 

11.  SPONSOR/MONITOR’S  REPORT 
NUMBER(S) 


Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std  Z39-18 


computer  platforms  and  to  facilitate  incorporation  within  fleet  TDAs.  Development  of  new 
sensor/system  models  is  expedited  through  code  reuse  and  sharing  of  common  interfaces,  making  it 
possible  to  leverage  off  previously  implemented  and  validated  models. 

APPROACH 

Performance  predictions  for  underwater  EOID  systems  rely  on  high-fidelity,  physics-based  simulation 
models.  These  models  account  for  the  specific  design  of  the  system,  and  include  detailed  sub-models 
for  the  laser  source,  the  receiver,  and  their  geometric  configuration.  Models  for  the  environmental 
conditions,  such  as  ambient  light,  and  a  complete  characterization  of  the  optical  medium  are 
incorporated  into  the  EODES  code  base.  The  light  distribution  in  the  water  column  from  all  light 
sources  (laser  and  ambient)  and  the  collection  of  backscattered  and  reflected  light  are  accurately 
modeled.  This  requires  solutions  to  the  radiative  transfer  equation  in  a  turbid  medium,  which  we 
assume  to  be  plane-stratified.  We  make  the  assumption  of  small-angle  scattering  and  use  well-known 
Fourier  optics  techniques  to  solve  for  the  light  distribution  and,  then,  to  derive  the  received  return. 

For  both  the  ELS  and  STIL  sensors,  along-track  scanning  is  achieved  by  the  forward  motion  of  the 
sensor  platform.  For  the  ELS  system  the  laser  source  is  mechanically  scanned  in  the  cross-track 
direction.  For  the  STIL  system  the  light  is  emitted  in  a  fan  beam,  and  the  received  light  is  processed 
and  converted  to  a  row  of  grayscale  pixel  values.  A  grayscale  raster  image  of  the  bottom  scene  is  built 
up  as  the  sensor  moves  forward  in  the  along-track  direction.  We  have  developed  accurate  simulations 
of  these  processes  for  the  LLS  and  STIL  systems  in  order  to  generate  realistic  synthetic  images.  These 
simulations  require  accurate  low-level  physical  models  that  form  a  common  code  base  for  all  system 
simulation  models,  and  which  also  form  the  basis  for  the  performance  prediction  models. 

Performance  predictions  can  be  based  directly  on  simulated  imagery  or  on  a  parametric  equation  that 
reflects  the  image  formation  process.  Simulated  images  provide  a  qualitative  and  subjective  means  of 
evaluating  performance  under  prescribed  conditions.  Parametric  models  provide  objective, 
quantitative  outputs  that  can  be  a  measure  of  relative  performance,  a  task  satisfaction  confidence  level, 
or  a  task-specific  probability  of  performance  measure.  EODES  uses  a  parametric  performance  model 
that  first  estimates  the  image  quality  under  specified  environmental  conditions,  and  then  relates  the 
image  quality  estimate  to  operator  performance  through  semi-empirical  models  based  on  actual  system 
imagery  and  operator  performance. 

The  EODES  image  quality  metric  predicts  the  expected  performance  of  an  EOID  system  operating 
under  given  environmental  conditions  and  with  specified  operational  settings.  The  metric  does  not 
require  the  generation  of  simulated  images.  We  scale  the  metric  to  range  between  0  and  10,  where  0 
indicates  a  complete  inability  to  identify  objects  within  the  imagery  obtained  by  the  system,  and  10 
indicates  an  excellent  ability  to  identify  objects.  The  EODES  metric  accounts  for  image  degradation 
due  to  limited  resolution,  blurring,  contrast  loss,  and  shot  noise.  It  is  patterned  after  the  General  Image 
Quality  Equation  (GIQE)  [1,2]  and  takes  the  form 

IQ  =  min  (l 0,  A  logj^  {SNR  +  B))  -  C  max  (O,  GSD  -  D  )  -  E  Vl  -  RER , 

where  A,  B,  X,  A,  and  E  are  system-dependent  coefficients  that  are  derived  based  on  an  analysis  of 
actual  system  data.  The  EODES  image  quality  equation  adapts  the  GIQE  to  account  for  the  fact  that 
underwater  images  are  obtained  from  relatively  short  standoff  distances. 
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As  with  the  standard  GIQE,  the  above  image  quality  metric  involves  three  measures  of  image  fidelity. 
The  ground  sampled  distance  (GSD)  is  a  measure  of  the  resolution  of  the  system  in  meters  at  the  depth 
of  the  scene  (ocean  bottom).  The  relative  edge  response  (RER)  is  a  measure  of  blurring  due  to  the 
system  optics,  geometric  spreading  of  the  narrow,  collimated  laser  beam  and  the  collector  field-of- 
view,  and  volume  scattering  in  the  water  column.  The  signal-to-noise  ratio  (SNR)  relates  the  fraction 
of  light  that  has  reflected  from  a  resolution  cell  on  the  ocean  bottom  (i.e.,  the  scene)  to  the  total  amount 
of  light  reaching  the  collector,  which  includes  volume  backscatter  of  both  ambient  and  laser  light. 

WORK  COMPLETED 

An  EOID  performance  prediction  model  was  developed  based  on  the  GIQE  formulation.  While  the 
new  EODES  model  depends  on  the  same  mid-level  variables  (the  GSD,  RER,  and  SNR),  the  EODES 
model  provides  a  new  definition  for  the  SNR  and  involves  a  very  different  functional  form.  This 
model  was  validated  against  ELS  data  collected  during  an  ONR  sponsored  field  test  conducted  in 
August  2001  near  Panama  City  Beach,  Florida  [4]. 

A  stand-alone  version  of  the  ELS  performance  prediction  model  (called  EODES  EES)  was  developed 
for  submission  to  GAME,  and  was  demonstrated  at  the  RIMP AC-08  exercise  conducted  in  Hawaii  in 
July  2008.  This  model  incorporates  the  metric  described  above  and  can  operate  in  one  of  two  modes:  it 
can  predict  image  quality  given  the  system's  operational  settings  and  a  characterization  of  the  optical 
environment;  or  it  can  determine  the  maximum  altitude  at  which  the  system  is  expected  to  produce 
imagery  suitable  for  the  purpose  of  target  identification. 

The  EODES  EES  code  was  submitted  in  August  2008  to  GAME  along  with  complete  software  and 
technical  documentation  and  a  comprehensive  test  suite.  The  documentation  includes  a  detailed 
application  programming  interface  (API)  description,  a  software  requirements  description,  and  a 
software  test  description.  The  EODES  EES  code  is  written  in  ANSI  C/C++  and  is  portable  across 
computer  platforms  (it  was  made  ISO  compliant  for  compilation  on  Windows  platforms). 

Extensive  validation  of  several  EODES  components  was  performed  in  collaboration  with  Dr.  Fraser 
Dalgleish  at  Harbor  Branch  Oceanographic  Institution  /Florida  Atlantic  University  (HBOI/FAU). 
Experimental  data  acquired  under  carefully  designed  and  controlled  experiments  validated  the  EODES 
radiative  transfer  model  and  the  EODES  laser  backscatter  model  for  several  sensor  configurations  and 
across  a  wide  range  of  turbidity  levels. 

RESULTS 

The  EODES  radiative  transfer  solver  computes  the  beam  spread  function  (BSF)  under  the  small-angle 
scattering  assumption  [3].  The  BSF  is  a  fundamental  quantity  underlying  the  simulation  and 
performance  prediction  models.  Accurate  calculation  of  the  BSF  is,  therefore,  critical  to  the  reliability 
of  all  the  EOID  system  models.  Extensive  laboratory  measurements  of  the  BSF  were  taken  at  the 
Harbor  Branch  Oceanographic  Institution  (HBOI)  campus  of  Florida  Atlantic  University  (FAU)  for 
several  water  turbidities  and  at  several  different  ranges.  An  example  of  the  validation  results  is  given 
in  Figure  1 .  It  demonstrates  very  good  agreement  between  the  measured  and  computed  BSF  as  a 
function  of  distance  from  the  beam  axis,  where  the  beam  attenuation  coefficient  c  =  0.4  m''  and  the 
radiometer  was  placed  at  a  range  of  7  meters  from  the  laser  source. 
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Comparison  ol  Beam  Spread  Fundion  -for  c=0.40/m  a1  z=7m 


Figure  1:  Comparison  of four  experimental  measurements  of  the  BSF  with  an  analytical  model 
based  on  small-angle  scattering.  The  BSF  is  plotted  on  a  log-log  scale  as  a  function  of  radial 

distance  from  the  beam  axis. 


For  underwater  EOID  imaging  systems,  a  major  contributor  to  the  noise  level  is  the  volume  backscatter 
from  the  laser  beam  as  it  propagates  through  the  water  column.  Backscattered  light  does  not  interact 
with  the  bottom  scene  and  hence  carries  no  useful  information.  An  accurate  estimate  of  the  laser 
backscatter  is  required  to  compute  the  signal-to-noise  ratio  (SNR),  which  is  a  key  term  in  the  image 
quality  equation  above.  Therefore,  the  overall  reliability  of  the  simulation  and  performance  prediction 
models  depends  on  an  accurate  estimate  of  the  laser  volume  backscatter.  This  computation  has  been 
validated  against  experiments  conducted  at  HBOI/FAU.  The  experiments  involved  sending  short  laser 
pulses  into  turbid  water  and  measuring  the  intensity  of  the  backscattered  light  as  a  function  of  time. 
Figure  2  is  an  example  of  the  results  obtained  when  comparing  the  model  calculations  to  the  measured 
data.  In  this  case  the  beam  attenuation  coefficient  c  =  0.25  m  '  and  the  source  and  receiver  separation 
was  set  at  376  mm.  The  second  peak  in  the  figure  is  due  to  the  reflection  from  a  panel  placed  at  10 
meters  from  the  source.  The  agreement  between  the  backscatter  model  and  the  data  is  excellent. 

In  most  MCM  situations  the  EOID  system  is  power  limited,  so  an  accurate  estimate  of  the  SNR  term  in 
the  image  quality  equation  is  critical  to  reliable  performance  prediction.  Figure  3  shows  the  predicted 
SNR  for  data  collected  at  an  ONR-sponsored  field  exercise  conducted  by  NSWC  in  Panama  City 
Beach  (PCB),  Florida,  in  August  2001  [4].  The  SNR  is  plotted  against  a  measure  of  towbody  altitude, 
which  is  given  in  attenuation  lengths  plus  the  natural  log  of  the  altitude  in  meters.  This  dimensionless 
distance  metric  is  roughly  proportional  to  the  power  received  as  a  function  of  distance.  During  the 
exercise,  the  platform  was  towed  at  several  different  altitudes  in  water  of  varying  turbidity.  Each  of 
the  ELS  runs  during  the  weeklong  exercise  was  color-coded  according  to  whether:  all  the  targets  in  the 
imagery  could  be  confidently  identified  (green);  many  targets  could  be  identified  (light  blue);  roughly 
half  the  targets  could  be  at  least  tentatively  identified  (blue);  only  a  small  fraction  of  targets  could  be 
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identified  (pink);  or  no  targets  were  identifiable  (red)  by  a  human  operator.  The  plot  shows  that  the 
estimated  SNR  is  itself  a  reasonably  good  discriminator  of  performance,  where  we  observe  that  the 
higher  the  SNR  the  greater  the  proportion  of  identifiable  targets. 


9/27/08  I  c=0.25/m  |  SR=376mm  |  SOD=7m  |  Aperture=3mtn  |  Angle=0mnad 


Figure  2:  Comparison  of  experimental  data  to  the  EODES  semi-analytical  model  for  laser  light 
backscattered  as  a  function  of  time.  A  pulse  is  emitted  at  time  t=0,  produces  backscatter  as  it 
propagates  through  turbid  water,  and  reflects  from  a  target  at  10  meters. 


Finally,  we  examine  the  predictions  for  image  quality  as  estimated  by  the  above  model  equation.  In 
Figure  4,  estimated  image  quality  is  plotted  against  the  same  metric  described  above.  We  see  that  the 
image  quality  metric  is  clearly  an  effective  discriminant  of  performance.  We  observe  that  horizontal 
lines  at  IQ=1  and  70=4  provide  very  good  separation  of  the  data  into  cases  where  all  targets  are  readily 
identifiable  (green),  where  identification  of  some  but  not  all  of  the  targets  is  possible  (light  blue,  blue, 
and  pink),  and  where  no  targets  are  identifiable  (red).  The  predictions  were  made  for  two  different 
LLS  systems  under  a  variety  of  environmental  conditions,  so  that  the  approach  is  applicable  across 
various  LLS  configurations  and  specifications.  The  color-coded  data  demonstrate  a  clear  and  strong 
correlation  between  the  predicted  image  quality  and  the  ability  of  the  operator  to  identify  targets  from 
within  the  imagery.  It  is  envisioned  that  such  predictions  of  EOID  performance  over  the  entire 
battlespace  will  aid  commanders  and  mission  planners  in  effectively  choosing  the  operational  areas, 
systems,  and  tactics. 
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Data  for  Laser  Line  Scan  Systems  Tested  in  Panama  City  Beach,  August  2001 


Towbody  Altitude  in  Attenuation  Lengths  +  log(Altitude) 


Figure  3:  Predicted  SNR  for  the  LLS  data  recorded  in  Panama  City  Beach,  Florida,  in  August 
2001.  Green  dots  (•)  indicate  that  all  targets  were  readily  identifiable  in  the  imagery,  blue 
squares  (m)  indicate  that  roughly  half  the  targets  were  identifiable,  and  red  diamonds  (*) 
indicate  no  identifiable  targets.  The  pink  (m)  and  light  blue  (m)  squares  indicate  runs  where  most 
of  the  targets  were  identifiable  or  unidentifiable,  respectively. 


Data  for  Laser  Line  Scan  Systems  Tested  in  Panama  City  Beach,  August  2001 
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Towbody  Altitude  in  Attenuation  Lengths  +  log(Altitude) 


Figure  4:  Predicted  image  quality  for  the  LLS  data  recorded  in  Panama  City  Beach,  Florida,  in 
August  2001.  Refer  to  the  caption  of  Figure  3  for  a  description  of  the  color-coded  markers. 
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IMPACT/APPLICATIONS 


When  completed  and  validated,  the  performance  prediction  models  for  LLS,  STIL  and  ALMD  systems 
will  be  available  for  distribution  and  incorporation  into  Fleet  Tactical  Decision  Aids.  These  models 
will  facilitate  effective  MCM  mission  planning,  tactical  decision  making,  and  asset  allocation. 
Moreover,  the  models  can  be  used  to  maximize  system  performance  by  estimating  optimal  system 
settings  for  a  given  environment.  The  BODES  model  suite  can  also  generate  synthetic  images  which 
are  valuable  for  training  human  operators,  and  can  also  be  useful  in  the  development  and  validation  of 
automatic  target  recognition  algorithms.  The  system  models  and  radiative  transfer  solvers  within 
BODES  will  also  become  useful  design  and  analysis  tools  for  prospective  electro-optical  systems. 

TRANSITIONS 

The  models  and  metrics  developed  and  validated  by  this  work  have  been  used  to  support  MIREM 
exercises  in  2003  and  2005  and  the  RIMPAC  exercises  in  2006  and  2008,  all  of  which  involved  the 
AQS-24  system.  Specifically,  for  the  RIMP AC-08  exercise,  the  EODES  performance  prediction 
model  was  a  key  part  of  the  Naval  Oceanographic  Office  (NAVOCEANO)  ocean  optics  and  mine 
warfare  demonstration.  The  EODES  models  are  currently  in  the  process  of  obtaining  OAML 
certification  in  preparation  for  transition  to  the  Naval  Oceanographic  Office  and  integration  into  the 
MEDAL/EPMA  tactical  decision  aids. 

RELATED  PROJECTS 

This  project  is  closely  related  to  award  number  N00014-08-M-0007,  which  provided  support  for 
RIMP  AC-08,  a  Naval  exercise  conducted  in  June  2008  in  Hawaii  where  we  demonstrated  how  systems 
for  battlespace  awareness  can  work  in  conjunction  with  EOID  performance  prediction  models  to 
provide  value  to  MCM  mission  planners.  The  same  award  number  also  provided  support  for  readying 
EODES_LLS  for  submission  to  the  OAML  certification  process. 

Award  N00014-04-C-0240  provided  support  for  experimental  validation  work  at  FAU/HBOI. 
Measurements  taken  there  of  the  beam  spread  function  and  laser  volume  backscatter  were  essential  to 
demonstrating  the  reliability  of  several  components  of  EODES  and  to  preparing  the  LLS  model  for 
OAML  accreditation  as  a  Navy  Standard. 
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